Consume AI model catalog from middlecache#30505
Conversation
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
|
Preview URL: https://84d4f2b7.preview.developers.cloudflare.com |
|
/bonk please review |
There was a problem hiding this comment.
A few issues to address before merge.
Out-of-scope notes (not blocking):
bin/compare-model-sources.tsreferences the deleted local JSON directories. Since model data now lives in middlecache, this utility is dead code and should be removed or rewritten in a follow-up PR.AGENTS.mdstill references the deletedworkers-ai-modelscontent collection. Update it to reflect the newai-catalogandworkers-ai-catalogcollections.
|
Review complete. No build breakers found ( Issues:
Out-of-scope notes (not blocking):
|
|
/bonk please review |
There was a problem hiding this comment.
A few minor issues to address.
Out-of-scope notes (not blocking):
- Cases 1 and 2 in ModelDetailPage.astro (lines 460–561) are now dead code because always sets and . With the middlecache path, only Cases 3 and 4 (parametersFlat / parametersMulti) ever execute. Consider removing the legacy build-time schema paths in a follow-up PR.
- still references deleted local JSON directories — should be removed or rewritten for middlecache.
There was a problem hiding this comment.
A few minor issues to address.
Out-of-scope notes (not blocking):
- Cases 1 and 2 in ModelDetailPage.astro (lines 460-561) are now dead code because detailToResolved always sets schema to empty objects and apiModes to undefined. With the middlecache path, only Cases 3 and 4 (parametersFlat / parametersMulti) ever execute. Consider removing the legacy build-time schema paths in a follow-up PR.
- bin/compare-model-sources.ts still references deleted local JSON directories -- should be removed or rewritten for middlecache.
|
Review complete. Out-of-scope notes (not blocking):
|
|
/bonk please review |
|
Review complete. All previously flagged issues have been addressed ( Issues:
Out-of-scope notes (not blocking):
|
…meters - Fetch parameters.json at build time in ModelDetailPage.astro frontmatter - SchemaDisplayFromRows.astro renders pre-processed SchemaRowData using SchemaTreeView/SchemaVariantSelector (pure Astro, no React component) - ModelDetailPage uses four clean separate conditional blocks instead of one ternary chain — Starlight Tabs used correctly for flat Input/Output, matching production layout exactly - Delete SchemaDisplayLazy.tsx (React component no longer needed)
Plain <h2 id="..."> elements inside StarlightPage slots don't pass through the rehype pipeline, so they never get the .heading-wrapper anchor link structure. Replace all section-level h2s with AnchorHeading (same pattern used by WranglerCommand, CompatibilityFlags, etc.).
Replace repeated Open/Download Lucide SVGs in ModelDetailPage with a dedicated SchemaFileLinks.astro component.
Replace Date.now() with a stable hash of the schema JSON so sessionStorage expand/collapse state persists across builds.
ModelDetailPage.astro now imports ModelInfo.astro (pure display, no interactivity). ModelInfo.tsx kept for use inside ModelCatalog.tsx (client React context).
Fixes any types, fragment-in-tr HTML bug, and locale-sensitive Intl formatting that could differ between build and client.
…ayout Cases 1 and 4 (build-time schema and pre-processed rows) now share the same MultiModeParams component for the shared-input/differing-input collapsible structure.
…psible layout" This reverts commit f373f62.
/workers-ai/glossary/ → /workers-ai/platform/glossary/
- model-resolver.ts: filter .rows.json from schemaFiles defensively (worker proxy only serves raw .json schema files) - ModelDetailPage.astro: clarify Cases 1 and 2 are legacy/unused paths now that detailToResolved always sets empty schema/apiModes - ModelFeatures.astro: add trailing slash to /workers-ai/function-calling/ - bin/compare-model-sources.ts: delete dead code referencing deleted local JSON directories - AGENTS.md: replace deleted workers-ai-models collection with new ai-catalog and workers-ai-catalog middlecache-backed collections
- bin/fetch-models.ts: downloads models.tar.gz from middlecache and extracts to .tmp/middlecache/v1/workers-ai-model-catalog/models/ (same pattern as bin/fetch-skills.ts) - package.json: wire into prebuild and predev hooks - ModelDetailPage.astro: use downloadToDotTempIfNotPresent for parameters.json — reads from .tmp/ if already extracted by fetch-models.ts (no-op), otherwise fetches from middlecache HTTP
Allows triggering a production rebuild programmatically via the GitHub Actions API without needing to push a commit.
…fetch-models - getDotTmpPath() in model-resolver.ts resolves .tmp/ relative to src/util/ (repo root) — ModelDetailPage.astro was resolving from src/components/models/ giving src/.tmp/ instead of .tmp/ - fetch-models.ts now clears stale catalog files on each run so all-models-detail.json etc. are always re-fetched after a pipeline graduation
- [fetch-models] and [fetch-skills] blue prefix tags via kleur - dim text for skip message, cyan for re-fetch hint - fetch-models: move stale cache cleanup before download so downloadToDotTempIfNotPresent always re-fetches after --force
- ModelCatalog.astro: server-renders all cards as static HTML with data-* attributes; vanilla JS <script> owns filter/sort/URL state - FilterDropdown.tsx + FilterDropdownWrapper.tsx: extracted from old ModelCatalog.tsx; wrapper self-initialises from URL params on mount and communicates via model-catalog-filter CustomEvents - SortSelect.tsx + SortSelectWrapper.tsx: same pattern for sort order - ModelBadges.astro: direct port of ModelBadges.tsx (no interactivity) - ModelInfo.astro: widened Props to accept ModelCardData | ResolvedModel - Deleted ModelCatalog.tsx
- FilterDropdown: remove CustomEvent dispatch (FilterDropdownWrapper owns it — both were firing on every selection causing applyFilters to run twice) - SortSelect: same fix — remove duplicate dispatch - FilterDropdown: remove unused filterKey prop (only the wrapper needs it) - ModelCatalog.astro syncUrl: only pushState when URL actually changes (previously pushed duplicate history entries on every filter pass, breaking browser back navigation) - Delete ModelFeatures.tsx (superseded by ModelFeatures.astro) - compare-model-sources.ts was already deleted in a prior commit
downloadToDotTempIfNotPresent was fetching from middlecache when the file wasn't found, but individual model parameters.json files aren't served as standalone HTTP endpoints (they only exist in models.tar.gz). In CI this caused every model detail page to get a 404 HTML response and silently render without a Parameters section. parameters.json is always extracted by bin/fetch-models.ts (prebuild); read it directly with readFileSync instead.
Previously ai-catalog.json, workers-ai-catalog.json, and all-models-detail.json were fetched lazily by downloadToDotTempIfNotPresent during the Astro build. Now fetch-models.ts pulls all four files (three JSONs + tarball) in parallel before the build starts, so the entire catalog payload is on disk and nothing depends on HTTP during the build.
…tags - All log/warn/error messages now start with the blue tag - Warnings are yellow, errors are red, success is green - Extracted tag into a const to avoid repetition
import.meta.url in getDotTmpPath() and downloadToDotTempIfNotPresent resolves relative to the *compiled* file location during astro build, which is dist/ — making ../../.tmp resolve to dist/.tmp/ instead of the repo root .tmp/ where fetch-models.ts actually writes files. process.cwd() is always the repo root regardless of where the compiled output lives.
5365f2f to
db0ac61
Compare
# Conflicts: # AGENTS.md # bin/fetch-ai-models.js # bin/fetch-catalog-models.ts # src/components/ModelCatalog.tsx # src/content.config.ts # src/content/catalog-models/alibaba-hh1-i2v.json # src/content/catalog-models/alibaba-hh1-t2v.json # src/content/catalog-models/alibaba-qwen3-max.json # src/content/catalog-models/alibaba-qwen3.5-397b-a17b.json # src/content/catalog-models/alibaba-wan-2.6-image.json # src/content/catalog-models/anthropic-claude-haiku-4.5.json # src/content/catalog-models/anthropic-claude-opus-4.6.json # src/content/catalog-models/anthropic-claude-opus-4.7.json # src/content/catalog-models/anthropic-claude-sonnet-4.5.json # src/content/catalog-models/anthropic-claude-sonnet-4.6.json # src/content/catalog-models/anthropic-claude-sonnet-4.json # src/content/catalog-models/assemblyai-universal-3-pro.json # src/content/catalog-models/bytedance-seedream-4.0.json # src/content/catalog-models/bytedance-seedream-4.5.json # src/content/catalog-models/bytedance-seedream-5-lite.json # src/content/catalog-models/google-gemini-3-flash.json # src/content/catalog-models/google-gemini-3.1-flash-lite.json # src/content/catalog-models/google-gemini-3.1-pro.json # src/content/catalog-models/google-imagen-4.json # src/content/catalog-models/google-nano-banana-2.json # src/content/catalog-models/google-nano-banana-pro.json # src/content/catalog-models/google-nano-banana.json # src/content/catalog-models/google-veo-3-fast.json # src/content/catalog-models/google-veo-3.1-fast.json # src/content/catalog-models/google-veo-3.1.json # src/content/catalog-models/google-veo-3.json # src/content/catalog-models/inworld-tts-1.5-max.json # src/content/catalog-models/inworld-tts-1.5-mini.json # src/content/catalog-models/minimax-hailuo-2.3-fast.json # src/content/catalog-models/minimax-hailuo-2.3.json # src/content/catalog-models/minimax-m2.7.json # src/content/catalog-models/minimax-music-2.6.json # src/content/catalog-models/minimax-speech-2.8-hd.json # src/content/catalog-models/minimax-speech-2.8-turbo.json # src/content/catalog-models/openai-gpt-4.1-mini.json # src/content/catalog-models/openai-gpt-4.1.json # src/content/catalog-models/openai-gpt-4o-transcribe.json # src/content/catalog-models/openai-gpt-5.4-mini.json # src/content/catalog-models/openai-gpt-5.4-nano.json # src/content/catalog-models/openai-gpt-5.4-pro.json # src/content/catalog-models/openai-gpt-5.4.json # src/content/catalog-models/openai-gpt-5.5.json # src/content/catalog-models/openai-gpt-5.json # src/content/catalog-models/openai-gpt-image-1.5.json # src/content/catalog-models/openai-gpt-image-2.json # src/content/catalog-models/openai-o4-mini.json # src/content/catalog-models/openai-tts-1-hd.json # src/content/catalog-models/openai-tts-1.json # src/content/catalog-models/pixverse-v5.6.json # src/content/catalog-models/pixverse-v6.json # src/content/catalog-models/recraft-recraftv4-pro-vector.json # src/content/catalog-models/recraft-recraftv4-pro.json # src/content/catalog-models/recraft-recraftv4-vector.json # src/content/catalog-models/recraft-recraftv4.json # src/content/catalog-models/runwayml-gen-4.5.json # src/content/catalog-models/vidu-q3-pro.json # src/content/catalog-models/vidu-q3-turbo.json # src/content/workers-ai-models/bart-large-cnn.json # src/content/workers-ai-models/gemma-3-12b-it.json # src/content/workers-ai-models/gemma-7b-it.json # src/content/workers-ai-models/hermes-2-pro-mistral-7b.json # src/content/workers-ai-models/kimi-k2.5.json # src/content/workers-ai-models/llama-2-7b-chat-fp16.json # src/content/workers-ai-models/llama-2-7b-chat-int8.json # src/content/workers-ai-models/llama-3-8b-instruct-awq.json # src/content/workers-ai-models/llama-3-8b-instruct.json # src/content/workers-ai-models/llama-3.1-70b-instruct.json # src/content/workers-ai-models/llama-3.1-8b-instruct-awq.json # src/content/workers-ai-models/llama-3.1-8b-instruct.json # src/content/workers-ai-models/meta-llama-3-8b-instruct.json # src/content/workers-ai-models/mistral-7b-instruct-v0.1.json # src/content/workers-ai-models/mistral-7b-instruct-v0.2.json # src/content/workers-ai-models/phi-2.json # src/content/workers-ai-models/sqlcoder-7b-2.json # src/content/workers-ai-models/uform-gen2-qwen-500m.json # src/pages/ai/models/index.astro # src/schemas/index.ts
…models # Conflicts: # bin/fetch-catalog-models.ts # package.json # src/components/models/ModelDetailPage.astro # src/components/models/ModelFeatures.tsx # src/content/catalog-models/alibaba-hh1-i2v.json # src/content/catalog-models/alibaba-hh1-t2v.json # src/content/catalog-models/alibaba-qwen3-max.json # src/content/catalog-models/alibaba-qwen3.5-397b-a17b.json # src/content/catalog-models/alibaba-wan-2.6-image.json # src/content/catalog-models/alibaba-wan-2.7-i2v.json # src/content/catalog-models/anthropic-claude-haiku-4.5.json # src/content/catalog-models/anthropic-claude-opus-4.6.json # src/content/catalog-models/anthropic-claude-opus-4.7.json # src/content/catalog-models/anthropic-claude-opus-4.8.json # src/content/catalog-models/anthropic-claude-sonnet-4.5.json # src/content/catalog-models/anthropic-claude-sonnet-4.6.json # src/content/catalog-models/anthropic-claude-sonnet-4.json # src/content/catalog-models/assemblyai-universal-3-pro.json # src/content/catalog-models/google-gemini-2.5-flash-lite.json # src/content/catalog-models/google-gemini-2.5-flash.json # src/content/catalog-models/google-gemini-2.5-pro.json # src/content/catalog-models/google-gemini-3-flash.json # src/content/catalog-models/google-gemini-3.1-flash-lite.json # src/content/catalog-models/google-gemini-3.1-flash-tts.json # src/content/catalog-models/google-gemini-3.1-pro.json # src/content/catalog-models/google-imagen-4.json # src/content/catalog-models/google-nano-banana-2.json # src/content/catalog-models/google-nano-banana-pro.json # src/content/catalog-models/google-nano-banana.json # src/content/catalog-models/google-veo-3-fast.json # src/content/catalog-models/google-veo-3.1-fast.json # src/content/catalog-models/google-veo-3.1.json # src/content/catalog-models/google-veo-3.json # src/content/catalog-models/inworld-tts-1.5-max.json # src/content/catalog-models/inworld-tts-1.5-mini.json # src/content/catalog-models/inworld-tts-2.json # src/content/catalog-models/minimax-hailuo-2.3-fast.json # src/content/catalog-models/minimax-hailuo-2.3.json # src/content/catalog-models/minimax-m2.7.json # src/content/catalog-models/minimax-m3.json # src/content/catalog-models/minimax-music-2.6.json # src/content/catalog-models/minimax-speech-2.8-hd.json # src/content/catalog-models/minimax-speech-2.8-turbo.json # src/content/catalog-models/openai-gpt-4.1-mini.json # src/content/catalog-models/openai-gpt-4.1.json # src/content/catalog-models/openai-gpt-4o-transcribe.json # src/content/catalog-models/openai-gpt-5.4-mini.json # src/content/catalog-models/openai-gpt-5.4-nano.json # src/content/catalog-models/openai-gpt-5.4-pro.json # src/content/catalog-models/openai-gpt-5.4.json # src/content/catalog-models/openai-gpt-5.5-pro.json # src/content/catalog-models/openai-gpt-5.5.json # src/content/catalog-models/openai-gpt-5.json # src/content/catalog-models/openai-gpt-image-1.5.json # src/content/catalog-models/openai-gpt-image-2.json # src/content/catalog-models/openai-o4-mini.json # src/content/catalog-models/openai-tts-1-hd.json # src/content/catalog-models/openai-tts-1.json # src/content/catalog-models/pixverse-v5.6.json # src/content/catalog-models/pixverse-v6.json # src/content/catalog-models/recraft-recraftv3.json # src/content/catalog-models/recraft-recraftv4-1-pro-vector.json # src/content/catalog-models/recraft-recraftv4-1-pro.json # src/content/catalog-models/recraft-recraftv4-1-utility-pro-vector.json # src/content/catalog-models/recraft-recraftv4-1-utility-pro.json # src/content/catalog-models/recraft-recraftv4-1-utility-vector.json # src/content/catalog-models/recraft-recraftv4-1-utility.json # src/content/catalog-models/recraft-recraftv4-1-vector.json # src/content/catalog-models/recraft-recraftv4-1.json # src/content/catalog-models/recraft-recraftv4-pro-vector.json # src/content/catalog-models/recraft-recraftv4-pro.json # src/content/catalog-models/recraft-recraftv4-vector.json # src/content/catalog-models/recraft-recraftv4.json # src/content/catalog-models/vidu-q3-pro.json # src/content/catalog-models/vidu-q3-turbo.json # src/content/catalog-models/xai-grok-stt.json # src/content/catalog-models/xai-grok-tts.json # src/util/model-resolver.ts # src/util/model-types.ts
Fetches AI model data from the
workers_ai_model_catalogmiddlecache pipeline instead of committed JSON files.What changed
Data sources
catalog-models+workers-ai-modelscontent collections (committed JSON) withai-catalog+workers-ai-catalogbacked bymiddlecacheLoaderall-models-detail.jsonModel detail pages
parameters.json(pre-processedSchemaRowDataper model, extracted frommodels.tar.gzbybin/fetch-models.ts)sync-input.jsonetc.) served from R2 via worker proxy at request time — no longer static build outputs/ai/models/**schema requests to middlecacheCatalog index pages
ModelCatalogconverted from a React island to Astro — cards are static HTML baked at build time, filter/sort/URL state managed by a vanilla JS script@base-ui/reactModelBadgesconverted from React to AstroInfra
workflow_dispatchadded topublish-production.ymlso middlecache graduations can trigger a docs rebuildbin/fetch-models.tspre-extractsmodels.tar.gzas apredev/prebuildhook;--forcere-fetches and clears stale cached catalog files